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DESCRIPTION 

IMAGE DATA DISTRIBUTION SYSTEM AND IMAGE DATA TRANSMITTING 
APPARATUS AND IMAGE DATA RECEIVING APPARATUS FOR THE SYSTEM 



Technical Field 

The present invention relates to an image data 
distribution system for distributing image data and an image 
data transmitting apparatus and image data receiving apparatus 
for the system. 



Background Art 

Conventionally, there have been proposed devices for 
displaying a stereoscopic image viewed from an arbitrary 
viewpoint using multiple viewpoint images taken by a plurality 
of cameras. One example of such a display device is shown 
in Fig . 16 . This device includes : an input unit 95 for setting 
the left and right eye viewpoints; a computer 91 connected 
to input unit 95 to generate image data that constitutes two 
images viewed from the left and right eye viewpoints; and 
a stereoscopic display 98 that is connected to computer 91 
to receive image data of the two images and makes stereoscopic 
display by synthesizing the data through an interior circuit 
thereof. 

Computer 91 comprises: a memory 92 for storing multiple 
viewpoint image data; a CPU 93 for implementing a process 
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of generating image data that constitutes two images; an 
input/output interface 94 connected to stereoscopic display 
98 and input unit 95 for controlling the input values from 
input unit 95 and the input and output of image data; and 
a bus 96 for mutually connecting CPU 93, memory 92 and 
input/output interface 94. 

From the multiple viewpoint image data stored in memory 
92, one desired set of image data is selected while its left 
and right eye viewpoints for implementing stereoscopic display 
are designated by using the input unit 95. In this case, it 
is judged whether the image data viewed from the left eye 
viewpoint is present in the multiple-view data. If the data 
is present, the image data is output through input/output 
interface 94 . If the data is not present, the image data that 
is viewed from the left eye viewpoint is generated by an 
interpolating process using the image data constituting a 
predetermined plurality of images taken from the viewpoints 
adjacent to the left eye viewpoint, and the generated data 
is output through input/output interface 94. A similar 
process is carried out from the right eye viewpoint. 

However, since the conventional technology needs high 
enough memory and a high-performance CPU, it suffers from 
the problem of being impossible for a terminal such as a mobile 
terminal , which has only limited memory and a CPU with a limited 
processing performance, to realize the technology. 
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Further, since it is assumed in the conventional 
technology that a special stereoscopic display is directly 
connected to the computer, no consideration is taken for 
efficient transmission of image data, so that it is impossible 
to deal with the case where various kinds of stereoscopic 
displays are connected through a network. 

The present invention has been devised under the above 
circumstances, it is therefore an object of the present 
invention to provide an image data distribution system, its 
image data transmitting apparatus and image data receiving 
apparatus, which enable observation of a stereoscopic image 
viewed from a viewpoint even with a mobile terminal or the 
like. 

Another object is to enable identical original image 
data to be used for observations of a stereoscopic image viewed 
from arbitrary viewpoints through various kinds of 
stereoscopic displays connected by a network while realizing 
efficient transmission of stereoscopic image data. 

Disclosure of Invention 

in order to achieve the above objects, the image data 
distribution system according to the present invention and 
its image data transmitting apparatus and image data receiving 
apparatus employ the following means. 

Specifically, an image data distribution system of the 
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present invention includes: a request information receiving 
means for receiving client' s request information transmitted 
by way of a network; a request information analyzing means 
for analyzing the request information received by the request 
information receivingmeans ; amultiple viewpoint image supply 
means which selects necessary image data from coded and stored 
multiple viewpoint image data of images taken by a plurality 
of cameras, based on viewpoint information from the request 
information analyzed by the request information analyzing 
means and decodes and supplies selected data; an image 
generating means which, based on image data supplied from 
the multiple viewpoint image supplying means , generates image 
data of an image viewed from a predetermined viewpoint in 
conformitywith the request information; an image synthesizing 
means for synthesizing a plurality of images data generated 
by the image generating means, based on display unit 
information from the request information; a coding means for 
coding image data synthesized by the image synthesizingmeans; 
a transmitting means for transmitting coded image data by 
the codingmeans to the network; a receivingmeans for receiving 
the coded image data via the network; a decoding means for 
decoding the coded image data received by the receiving means ; 
an image processing means for processing decoded image data 
by the decoding means so as to be displayable on a display 
means; the display means for displaying image data processed 
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by the image processing means; a request information input 
means for allowing input of the client' s request information; 
and a request information transmitting means for transmitting 
the request information to the network. 

Also, an image data transmitting apparatus of the present 
invention includes: a request information receiving means 
for receiving client's request information transmitted by 
way of a network; a request information analyzing means for 
analyzing the request information received by the request 
information receivingmeans; amultiple viewpoint image supply 
means which selects necessary image data from coded and stored 
multiple viewpoint image data of images taken by a plurality 
of cameras, based on viewpoint information from the request 
information analyzed by the request information analyzing 
means and decodes and supplies selected data; an image 
generating means which, based on image data supplied from 
the multiple viewpoint image supplying means, generates image 
data of an image viewed from a predetermined viewpoint in 
conformity with the request information; an image synthesizing 
means for synthesizing a plurality of images data generated 
by the image generating means, based on display unit 
information from the request information; a coding means for 
coding image data synthesized by the image synthesizing means ; 
and a transmitting means for transmitting coded image data 
by the coding means to the network. 
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Further, the image data distribution system of the 
present inventionmay further include a management information 
adding means for adding management information for enabling 
access to the image data of individual viewpoints and random 
access, to the multiple viewpoint image data. 

Moreover, the image data transmitting apparatus of the 
present inventionmay further include a management information 
adding means for adding management information for enabling 
access to the image data of individual viewpoints and random 
access, to the multiple viewpoint image data. 

Still, an image data distribution system of the present 
invention includes: a request information receiving means 
for receiving client's request information transmitted by 
way of a network; a request information analyzing means for 
analyzing the request information received by the request 
information receivingmeans; amultiple viewpoint image supply 
means for supplying multiple viewpoint image data; an image 
generating means which, based on viewpoint information from 
the request information analyzed by the request information 
analyzing means, receives input of necessary image data from 
the multiple viewpoint image supply means and generates image 
data of an image viewed from a predetermined viewpoint in 
conformity with the request information; an image synthesizing 
means for synthesizing a plurality of images data generated 
by the image generating means, based on display unit 
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information from the request information; a coding means for 
coding image data synthesized by the image synthesizing means ; 
a transmitting means for transmitting coded image data by 
the coding means to the network; a receiving means for receiving 
the coded image data via the network; a decoding means for 
decoding the coded image data received by the receiving means; 
an image processing means for processing decoded image data 
by the decoding means so as to be displayable on a display 
means; the display means for displaying image data processed 
by the image processing means; a request information input 
means for allowing input of the client' s request information; 
a request information transmitting means for transmitting 
the request information to the network; and a judgement means 
for judging whether the received image data is of 
two-dimensional image data or stereoscopic image data. 

Another image data receiving apparatus of the present 
invention includes: a receiving means for receiving coded 
image data by way of a network; a decoding means for decoding 
the coded image data received by the receiving means; an image 
processing means for processing decoded image data by the 
decoding means so as to be displayable on a display 
the display means for displaying image data processed by the 
image processing means; a request information input means 
for allowing input of request information of a client ; a request 
information transmitting means for transmitting the request 
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information to the network; and a Judgement means for Judging 
whether the received image data is of two-dimensional image 
data or stereoscopic image data. 

Next, an image data distribution system of the present 
invention includes: a request information receiving means 
for receiving client's request information transmitted by 
way of a network; a request information analyzing means for 
analyzing the request information received by the request 
information receivingmeans ; amultiple viewpoint image supply 
means for supplying multiple viewpoint image data; an image 
generating means which, based on viewpoint information from 
the request information analyzed by the request information 
analyzing means, receives input of necessary image data from 
the multiple viewpoint image supply means and generates image 
data of an image viewed from a predetermined viewpoint in 
conformitywiththe request information; an image synthesizing 
means for synthesizing a plurality of images data generated 
by the image generating means, based on display unit 
information from the request information; a coding means for 
ceding image data synthesized by the image synthesizing means ; 
a transmitting means for transmitting ceded image data by 
the codingmeans to the network; a receivingmeans for receiving 
the coded image data via the network; a decoding means for 
decoding the coded image data received by the receiving means ; 
an image processing means for processing decoded image 
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data by the decoding means so as to be displayable on a display 
means; the display means for displaying image data processed 
by the image processing means; a request information input 
means for allowing input of the client' s request information; 
a request information transmitting means for transmitting 
the request information to the network; and an identification 
information adding means for adding to the image data to be 
transmitted a piece of information that indicates whether 
the image data is of two-dimensional image data or stereoscopic 
image data. 

Further/ an image data transmitting apparatus of the 
present invention includes: a request information receiving 
means for receiving client' s request information transmitted 
by way of a network; a request information analyzing means 
for analyzing the request information received by the request 
information receiving means ; a multiple viewpoint image supply 
means for supplying multiple viewpoint image data; an image 
generating means which, based on viewpoint information from 
the request information analyzed by the request information 
analyzing means, receives input of necessary image data from 
the multiple viewpoint image supply means and generates image 
data of an image viewed from a predetermined viewpoint in 
conformity with the request information; an image synthesizing 
means for synthesizing a plurality of images data generated 
by the image generating means, based on display unit 
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information from the request information; a coding means for 
coding image data synthesizedby the image synthesizingmeans; 
a transmitting means for transmitting coded image data by 
the coding means to the network; and an identification 
information adding means for adding to the image data to be 
transmitted a piece of information that indicates whether 
the image data is of two-dimensional image data or stereoscopic 
image data. 

Moreover, an image data receiving apparatus of the 
present invention includes: a receiving means for receiving 
coded image data by way of a network; a decoding means for 
decoding the coded image data received by the receiving means; 
an image processing means for processing decoded image data 
by the decoding means so as to be displayable on a display 
means; the display means for displaying image data processed 
by the image processing means; a request information input 
means for allowing input of request information of a client; 
a request information transmitting means for transmitting 
the request information to the network; and an identification 
information adding means for adding to the received image 
data a piece of information that indicates whether the image 
data is of two-dimensional image data or stereoscopic image 
data. 
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As has been described heretofore, since the above 
configuration makes it unnecessary to provide large amounts 
of memory and a high-performance CPU on the client side, the 
invention has the effect of enabling observation of a 
5 stereoscopic image viewed from an arbitrary viewpoint even 

with a mobile terminal and the like. 

The present invention also has the effect of enabling 
use of an identical original image data to provide observations 
of a stereoscopic image viewed from arbitrary viewpoints 
10 through various types of stereoscopic displays that are 

connected by way of a network. 

Brief Description of Drawings 

Fig.l is a block diagram showing an embodiment of an 
15 image data distribution system according to the present 

invention. 

Fig. 2 is an illustration showing layouts of a plurality 
of cameras for preparing multiple-view image data. 

Fig. 3 is an illustration showing the left eye viewpoint 
20 L and the right eye viewpoint R for which image data is generated 

by interpolation. 

Fig. 4 is an illustration showing areas to be cut out 
from the generated image data, depending on the resolution 
of a display unit. 
25 Fig . 5 is an illustrative view showing a mobile terminal 
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to be a client. 

Fig. 6 is an illustration showing examples of states of 
stored data of multiple-view image data. 

Fig. 7 is an illustration showing examples of states of 
generated image data. 

Fig. 8 is a diagram showing storage and extraction of 

multiple-view image data. 

Fig. 9 is a flowchart showing the sequence of processing 

on a server. 

Fig. 10 is a flowchart showing the sequence of processing 
on a client. 

Fig. 11 is an illustration showing an example of coding 
multiple viewpoint moving picture data based on MPEG-4. 

Fig. 12 is a diagram showing multiple viewpoint image 
data added with management information. 

Fig. 13 is a chart showing one example of management 
information. 

Fig. 14 shows an expected connection state between a 
server and clients in the present embodiment. 

Fig. 15 is a flowchart showing details of the processing 
of an image generator. 

Fig. 16 is a diagram showing a conventional example. 

Best Mode for Carrying Out the Invention 

The embodiment of an image data distribution system 
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according to the present invention will be described with 
reference to the drawings. 

Fig.l is a block diagram showing the embodiment of an 
image data distribution system according to the present 

5 invention. 

A server 1 (image data transmitting apparatus) and a 
client 11 (image data receiving apparatus) are connected to 
eachotherbywayof anetwor k 7. Server 1 has stored (recorded) 
multiple viewpoint image data 2. When client 11 transmits 
10 request information to server 1 , the client is able to implement 

stereoscopic display of an image viewed from a desired 
viewpoint on a display unit 14. 

Here, the multiple viewpoint image data has not been 
necessarily stored (recorded) previously in server 1, but 
15 may be supplied in real time from without. 

Server 1 analyses the request information transmitted 
from client 11 by a request analyzer 4 (request information 
analyzing means (including request information receiving 
means)) and selects the necessary image data from multiple 
20 viewpoint image data 2 (multiple viewpoint image supply means) 

to output it to an image generator 3 (image generating means) 
where image data for the requested viewpoint (viewpoint 
information) is generated by interpolation to be output to 
an image synthesizer 5 (image synthesizing means) . In image 
25 synthesizer 5, a plurality of supplied images data are 
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synthesized in a form (format based on the display unit 
information) suitable for enooding to be output to an encoder 

6 (coding means) . In encoder 6, the supplied image data is 
encoded at a suitable bit rate to be transmitted to network 

7 (transmitting means) . 

Client 11 receives the coded image data (receivingmeans) , 
and decodes the data through a decoder 12 (decoding means) 
and outputs the decoded image data to an image processor 13 
(image processing means, , where the image data is converted 
into an appropriate form in conformity with a stereoscopic 
display format so that the image data is displayed on a display 
unit 14 (display means) . Client 11 also includes an input 
unit 16 (request information input means) for change of the 
viewpoint, and transmits the reguest information of viewpoint 
alternation to network 7 by way of a request output unit 15 
(request information transmitting means) . 

Now, server 1 and client 11 will be described in detail. 
Multiple viewpoint image data 2 is formed of a set of 
images data taken by a plurality of cameras. The plurality 
of cameras are typically laid out as shown in Fig. 2 (a) so 
that the optical axes cf the plurality of cameras intersect 
atonepoint. As a special example, the camerasmaybe arranged 
on the circumference of a circle so that the optical axes 
of the cameras are directed to the center of the circle, as 
shown in Fig.2 (b) . In either case, the cameras are not 
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necessarily arranged equi-distantly. but may be laid out in 
some parts densely and others sparsely. The information as 
to how the cameras are allocated is also recorded together 
with the image data. This allocation information is used to 
determine the image data of which cameras should be used when 
image generator 3 generates the image data from a designated 
viewpoint by interpolation based on the request information 

from client 11. 

Next, description will be made about the necessary camera 
image data when the image of the requested viewpoint is 
generated by interpolation. In the example shown in Fig. 3 
(a) , the left eye viewpoint L and right eye viewpoint R are 
designated as illustrated with respect to cameras CI to C4. 
in this case, the images of data from CI and C2 are used to 
generate the image data for left eye viewpoint L and the images 
of data from C2 and C3 are used to generate the image data 
for right eye viewpoint R. Similarly, in the example shown 
in Fig. 3 ,b> , the layout of cameras CI to C4 and the requested 
left eye viewpoint L and right eye viewpoint R are positioned 
as illustrated. In this case, both the image data for left 
eye viewpoint L and the image data for right eye viewpoint 
r are generated based on the images of data from CI and C2. 
Though four cameras are used in the examples in Fig. 3, the 
number of cameras is not limited to four. 

Generation of images from an intermediate viewpoint by 
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interpolation is a known technology, and is disclosed in detail 
in, for example, "Tsunashima et al.: generation of 
intermediate image data from a two-view stereoscopic image 
data taking occlusion into consideration, 3D Image Conference 
'95, PP- 174-1-77 (1995)", and "Azuma et al. : parallax 
estimation based on the edge information for generation of 
intermediate image data, 3D Image Conference '95, pp. 190-194 

(1995)", and others. 

When the necessary viewpoint image data has been 
generated at image generator 3, image synthesizer 5 implements 
an extraction process of image data in an amount for the 
requested resolution. In Fig. 4 (a) , the size of the generated 
image data 21 is assumed to be equal to the size of the image 
taken by the camera, and the resolution required for display 
on client 11 is shown by an area 22. In Fig. 4 (a) , only part 
of the generated image data is cut out. Alternatively, as 
shown in Fig. 4 (b) , instead of cutting out an image in the 
resolution required for display on client 11, the maximum 
area 24 capable of being cut out while keeping the display 
aspect may be cut out from the generated image data 23 (which 
is assumed to be the same size as the image taken by the camera) , 
then may be reduced to the required resolution. Also, image 
generator 3 may be adapted to generate only the necessary 
amount of image data for the required resolution. 

Fig. 5 is an illustrative view showing a mobile terminal 
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50 to be a client 11. The terminal includes a stereoscopic 
display 54, a four-way arrow key 51 for moving the viewpoint 
up, down, left and right and keys 52 and 53 for moving the 
viewpoint forward and backward. Though unillust rated, the 
terminal also has a communication means for making 
communication with the server. 

Fig. 6 shows examples of storage states of multiple 
viewpoint image data 2. Because moving picture data taken 
by a plurality of cameras need to be temporally synchronized 
to each other, the plurality of images data CI to C4 taken 
by different cameras may be stored in such a manner as to 
join them abreast and form a piece of image data made of one 
image, as shown in Fig. 6 (a) . This format assures that the 
images data Cl to C4 contained in the image data of the single 
image were taken at the same time, leading to easy time 
management . The way of j oining is not limited to a line abreast 
as shown in Fig. 6 (a) but the images may be joined as shown 
in Fig. 6 (b) , for example. 

in contrast, as shown in Fig. 6 (c) , a format of separately 
storing the images of data Cl to C4 taken by different cameras 
m aybe also considered. The advantage of this method is that, 
when camera images Cl and C2 alone are needed as shown in 
Fig. 3 (b) in order to generate the image data from the requested 
viewpoint, these can be easily picked up. 

Multiple viewpoint image data 2 may be stored either 
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by being compressed or non-compressed. Here, referring to 
Fig. 8, description will be made about a case where individual 
pieces of camera image data which are given separately in 
the manner as shown in Fig. 6 (c) are stored after being data 
compressed. In this case, images of data Cl to C4 taken by 
the individual cameras are input to an encoder 31 as shown 
in Fig. 8 (a) . Encoder 31 encodes each image data and outputs 
the necessary information (frame type, generation bit count, 
etc. ) for generating management information, to a management 
information generator 32. In a recorder 33, the coded image 
data and the management information are recorded as the storage 
data (management information adding means) . The detail of 
the management information and storage format will be 
described later. 

When the storage data which has been recorded in the 
xnanner shown in Fig. 8 (a) is used as the multiple viewpoint 
image data, the data needs to be decoded so as to be handled 
by image generator 3. In this case, as shown in Fig. 8 (b) , 
a selector 34 , in accordance with the request from the client, 
selects only the necessary image data from the stored data 
and outputs to a decoder 35 where the data is decoded, whereby 
it is possible to obtain the necessary and sufficient original 
image data (the image data to be used by image generator 3) . 
in this process , in order to extract the necessary part quickly, 
the management information recorded together with image data 
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is utilized. 

Fig. 7 shows examples of image data states generated by 
image synthesizer 5 . Though the left eye viewpoint image data 
L and the right eye viewpoint image data R can be encoded 
as separate pieces of image data as shown in Fig. 7 (b> , it 
is preferred that both images of data are joined side by side 
(or up and down) into a piece of synthesized image data formed 
of a single image then the joined image data is encoded. 
Formation of the image data composed of a single image as 
shown in Fig. 7 (a) is able to assure the synchronism between 
the left eye viewpoint image data L and the right eye viewpoint 
image data R, leading to easy time management . Particularly, 
when a coding scheme entailing frame skipping such as MPEG-4 
is used, it is possible to prevent occurrence of such a situation 
where, at a certain moment , the frame of the left eye viewpoint 
image data L is present whereas the frame of the right eye 
viewpoint image data R is not present. Further, since the 
image data is rate-controlled as a single image, both the 
left eye viewpoint image L and the right viewpoint image R 
can be kept substantially equal in image quality. When the 
images are encoded as separate pieces of image data, depending 
on the rate control result there occurs a case where on 
extraction of a frame at a certain moment the image quality 
of the left eye viewpoint image L is good while the image 
quality of the right eye viewpoint image R is bad. In such 



16 



a case, the resultant stereoscopic display presents poor 
quality. Thus, it is possible to improve the quality of 
stereoscopic display when the data is adapted to take the 
form as shown in Fig. 7 (a). 

Depending on the stereoscopic display format of a client 
11, there are some cases where the image data to be finally 
displayed on the display unit 14 takes a form of strips of 
left eye viewpoint image data L and strips of right eye viewpoint 
image data R being alternated with each other every line as 
shown in Fig. 7 (c) (for lenticular mode, parallax barrier 
mode and the like) . In such a case, however, it is preferred 
that the image data to be coded takes the form as shown in 
Fig. 7 (a). The reason is that if coding is performed on a 
block basis as in DCT, the image data having the form as shown 
in Fig. 7 (c) will present weak correlation between adjacent 
pixels, hence high spatial frequencies, producing poor 
compression efficiency. The same method can be applied for 
the cases where the number of viewpoints is greater than 2. 

When images of data for a plurality of viewpoints are 
joined to form a piece of image data made of a single image 
as shown in Fig. 7 (a) , it is impossible to know the difference 
between normal two-dimensional image data and stereoscopic 
image data from a format point of view. No problem will occur 
when real-time streaming is handled using the system shown 
in Fig.l because image data is transmitted in real time in 



accordance with the request from the client side. However, 
in a case where the thus transmitted image data has been once 
recorded locally on the client 11 side and is played afterward, 
it is impossible to distinguish whether it is of 
two-dimensional image data or stereoscopic image data. In 
order to prevent this, when any piece of image data having 
the form as shown in Fig. 7 (a) is recorded, a flag for 
identification indicating whether the image data is of 
two-dimensional image data or stereoscopic image data may 
and shouldbe added (identification information adding means) . 
This addition of the identification flag may be done either 
on server 1 or client 11. Further, client 11 has a judgement 
means for distinction between two-dimensional image data or 
stereoscopic image data. 

in connection with the above, when client 11 records 
image data locally, the recording means shouldbe placedbef ore 
decoding (upstream of decoder 12) or after image processing 
(downstream of the image processor) . 

Fig. 9 is a flowchart showing the sequence of processing 
on server 1. First, the request from a client is analyzed 
(Step SI) - Then, the necessary image data is selected from 
multiple viewpoint image data (Step 32) - Next, an image from 
the requested viewpoint is generated based on the selected 
data (Step S3) . Subsequently, the image data of a size 
required for display is cut out (and reduced as required) 
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(Step S4). Then, the cutout left eye viewpoint image data 
and the cutout right eye viewpoint image data are joined (Step 
S5). Next, the joined image data is encoded (Step S6) . 
Finally, the data is output as a bit stream (Step S7) . 

Fig. 10 is a flowchart showing the sequence of processing 
on client 11. First, initialization is done so that the 
viewpoint position (viewpoint information) at the initial 
state and information not dependent on the viewpoint 
(stereoscopic display format, resolution etc. (display unit 
information)) are set up (Step Sll) . Next, these pieces of 
information are transmitted as a request to server 1 (Step 

512) . Then, the bit stream (image data) meeting the request 
is transmitted from the server by way of the network (Step 

513) . Subsequently, the bit stream is decoded (Step S14) - 
Since, as shown in Fig. 7 (a) the decoded image data is not 
in the form that is directly and stereoscopically displayable, 
the data is rearranged so as to conform with the stereoscopic 
display format, as shown in Fig. 7 (c) (Step S15) . Then the 
data is displayed on display unit 14 (Step S16) . Next, it 
is judged whether there is a next display (Step S17) . If 
display is continued, it is judged whether there is a request 
for change of the viewpoint (Step S18) . Subsequently, if the 
viewpoint should be changed, the request is sent again to 
server 1 and the operation returns to Step S12 - When no more 
display is needed at Step S18, the operation goes to Step 
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S13. 

Fig. 14 shows an expected connection state between a 
server and clients in this embodiment. Terminals A to C are 
the clients and are connected to a server 41 via a network 
7. Each terminal has a stereoscopic display format and a 
display resolution different from others, and the position 
of the viewpoint for observation is different from others. 
Therefore, server 41, in accordance with the request from 
each terminal, transmits to the terminal different image data 
from the others . If the whole multiple viewpoint image data 
stored (recorded) on server 41 is transmitted to the network 
so as to permit each terminal to choose the necessary data 
to display, transmission can be done with one kind of image 
data. However, since the multiple viewpoint image data is 
composed of a massive amount of data and because of bandwidth 
limitations of the network, such a transmission is unfeasible . 
This is why the transmission system of the present invention 
in which the server can transmit appropriate image data in 
accordance with the request from a client is requisite to 
an environment in which different kinds of terminals are 
connected to a network. 

Fig. 15 is a flowchart showing details of the processing 
in the image generator 3 . First, the position of the viewpoint 
requested from a client is analyzed (Step S21) . Next, it is 
judged whether there is any image whose viewpoint corresponds 



to the requested viewpoint, among the stored multiple 
viewpoint image data (Step S22) . If there is, the image data 
is directly used; otherwise the interpolation image is 
generated (Step S23) . Subsequently, the data is output as 
the image data (Step S24) . 

Next, management information will be described. 
Fig. 11 shows one example of coding a multiple viewpoint 
moving picture based on MPEG-4 . Since frame skipping can be 
used with MPEG-4, the frames to be coded have discontinuous 
frame numbers such as LTfO, LTf3, LTf5 and LTflO. Further, 
in this example, LTfO is assumed to be coded as an intra frame 
coding frame (I-frame) , LTflO is assumed to be coded as a 
frame (P-frame) which is predictive coded based on the decoded 
frame of LTfO, and LTf 3 and LTf 5 are assumed to be coded as 
frames (B-frames) which are bi-directionally predictive coded 
based on the decoded frames of LTfO and LTflO. 

Fig. 13 is a chart showing one example of management 
information added at management information generator 32 shown 
in Fig. 8 (a) . The coded data of individual camera images can 
be joined to management information as shown in Fig. 12 and 
stored. This management information is the information that 
allows for access to each camera image data. In the case of 
a multiple viewpoint moving picture, the management 
information contains not only the information which allows 
for access to each camera image but also the information which 
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allows random access to the coded data at a designated time 
within each camera image data. 

Fig. 13 (a) shows one example of management information 
for access to the coded data of individual camera images. 
5 For example, it is indicated that the coded data of camera 

image C2 exists at the B2-th byte from the front of the data 
in Fig. 12. Fig. 13 (a) further includes the pointers to the 
information for making access to the coded data at designated 
times within the camera image data. For the coded data of 
10 C2, it is indicated that the access table for making access 

to the coded data at designated times is located at address 
P2 within the management information. 

Fig. 13 (b) shows one example of an access table to coded 
data at designated times. Times tl. t2, t3, ■•■ may be set 
15 up at regular intervals or may be set up arbitrary intervals 

apart. For example, it is indicated that the coded data 
corresponding to time t3 exists at the Bt3-th byte from the 
front of the coded data of the camera image while the coded 
data of I-frame is located at a position upstream by It3 bytes 
from the aforementioned position. If the decoder needs to 
start display from time t3, the coded data of the I-frame 
located at the (Bt3-It3)-th byte from the front is decoded 
first. Then, P-frames and B-frames are successively decoded 
while counting the number of bytes of the decoded data until 
25 the count reaches It3 bytes. When the display is started at 
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this point of time, the image data from the designated time 
t3 is displayed. 

Next, other accessing methods will be described. 
(A) coded data is packetized and the header information of 
each packet has information that indicates whether the packet 
contains the front of an I-£rame. In Fig.13 (b, , the 
designated time and the number of bytes to the packet 
corresponding to the designated time are written. When the 
decoder had made access to the packet corresponding to the 
designated time t3, it is checked as to whether the packet 
contains the front of an I-f rame. The decoder starts decoding 
and display from a packet that contains an I-frame (all the 
packets before that are discarded as unnecessary) . 
,B) m (A), instead of indicating the number of bytes to 
the packet, only the packet nurtoer is written in Fig. 13 (b) . 
The length of the packets in a piece of coded data is assumed 
to be fixed and the number of bytes of one packet is written 
in the header information of the coded data. The decoder 
calculates the nu^er of bytes to the packet corresponding 
to the designated time based on the packet number and the 
number of bytes of one packet (thereafter the steps are the 

same as (A) ) . 

next, other storage forms will be described. 
,C) in Fig. 12, the management information and the coded 
information are j0 ined and stored. However, the management 
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information may be separated and stored as a different file. 
,D) Of themanagement information, the information for access 
to designated times may be inoiuded in the header information 
of the coded data of each camera image , insteadof being included 
in themanagement information. Inthiscase, the third column 
in rig. 13 (a) (the pointer to the information for access to 
designated times within each camera image) is not necessary. 
(E , The management information, the coded data of individual 
camera images may be all separated into different files. I» 
this case, the nutftoer of bytes from the front in the second 
column in Fig. 13 (a, is replaced by the filename of the coded 
data of each camera image, for example. Access to each camera 
image is made based on the filename. 

Industrial Applicability 

The image data distribution system, its image data 
transmitting apparatus and image data receiving apparatus 
according to the present invention does not need large amounts 
of memory and a high-performance CPU on the client side, and 
can be applied to mobile terminals which enable observations 
of a stereoscopic image from arbitrary viewpoints. 



